Sensor fusion is a process of combining the sensory data derived from different sensors. As an increasing number of sensors and sensor modalities are used to acquire scenes, consolidation or fusion of the sensor data is becoming increasingly important. Sensor fusion exploits the distinct sensor modalities to provide complimentary information about the environment, overcome hardware limitations, or reduce data uncertainty due to each individual sensor. For example, the sensor fusion can increase, i.e., upsample, the resolution on data measured by one sensor using measurements of another sensor. Additionally or alternatively, the sensor fusion can annotate the data measured by one sensor with the measurements of another sensor.
For example, the depth sensing is a technology for measuring depths in a scene, i.e., the distances from a sensor to points in the scene. Types of depth sensing include measurements using structured light cameras, stereo cameras, and depth sensing cameras based on time-of-flight (TOF) measurements of the light reflected from the scene. Some depth sensors, such as LIDAR sensor, do not have sufficient resolution for practical applications. To that end, the fusion can be used for the depth superresolution, i.e., the low-resolution depth data from a LIDAR sensor can be fused with an image from an optical camera to produce a higher-resolution depth image.
The sensor fusion uses an extrinsic calibration that determines the calibration parameters of each sensor, such as position and orientation of each sensor with respect to each other. During the fusion, the calibration parameters are used to compute the geometric transformation that maps the output of each sensor to a common frame of reference.
For example, some methods perform offline calibration using known alignment targets. However, performing the offline calibration is not possible or practical for some applications. This problem is especially apparent when the sensors are installed at the moving vehicle. This is because such sensors are prone to lose the calibration due to potential roughness of the road conditions, and the calibrations needs to be performed online for constantly varying scenes.
Other methods perform calibration online using edges of the objects in the different images. However, in some applications, the calibration based on edge matching is inaccurate due to low resolution of the sensor measurements.
Accordingly, there is a need for a system and a method for fusing outputs of sensors having different resolution.